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ABSTRACT 

The influence of survey return rate and sources of 
information other than the survey on the teaching employment rate (as 
calculated) in the year following completion of a teacher preparation 
program were studied. A second objective was to investigate the 
tendency to respond for these individuals. A total of 291 individuals 
formed Che target population for the study following graduates of a 
teaching internship. Thirty-four percent responded to the initial 
survey, and 69.8% were teaching. The second mailing increased the 
response rate to 51%, and the cumulative percentage teaching to 
71.77.. The third request brought the response rate to 577. and the 
teaching rate to 73.67.. "Telephone calls and late responses increased 
response rate but depressed teaching rate. Adding information from 
other sources brought coverage to 937., and the percentage teaching to 
75.57.. A spring survey yielded similar results for nonrespondents . 
From a statistical point of view, early returns did not differ 
significantly from those achieved by subsequent efforts. Results also 
suggest that deciding to participate in a survey is highly dependent 
on the inclination of the individual. Two tables present study 
findings. (Contains 26 references.) (SLD) 
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Effect of Response Rate on Results of a Follow-up Employment Survey 

Nonresponse bias is considered a threat to the validity of survey findings unless nonrcsponse is 
slight or nonexistent. Nonresponse bias occurs when the results obtained from respondents do not 
accurately represent the group being surveyed because those who did not respond differ from the 
respondents and the survey results would have been different had the nonrespondents chosen to 
participate. Various ways of addressing the problem have been offered (Hartman, Fuqua, and Jenkins, 
1986). 

Survey researchers have been encouraged to strive for high mail survey return rates to insure that 
the respondents are representative of the group surveyed, and much of the research literature in survey 
methodology is devoted to increasing response rates. However, techniques used to promote higher 
response rates have sometimes induced or increased the amount of bias in the sample (Jones & Lang, 
1980). It is important, therefore, to determine the extent of the problem (bias resulting from nonresponse) 
before deciding to implement procedures that may create bias in and of themselves. According to 
Dillman (1991), "A low response rate docs not necessarily entail nonresponse error" (p. 229). He goes on 
to point out, "the usual reason that surveys are commissioned is that the distribution of certain 
population characteristics is unknown and a survey becomes the tool to find out the distribution of those 
characteristics." Thus it is ordinarily difficult to gauge the true extent of nonresponse bias, 

Some researchers have employed mathematical solutions to compensate for nonrcsponse (Aiken, 
1988), while others have investigated the extent of nonresponse bias by various approaches: comparing 
respondents with nonrespondents using data already on file; comparing respondents with a sample of 
nonrespondents who are interviewed to obtain their answers to the items on the questionnaire; 
comparing responses of early respondents with those of late respondents on the assumption that the late 
respondents arc similar to nonrespondents and would, in fact, have been nonrespondents had follow-up 
attempts not been instigated; and comparing early respondents with all respondents, The variables being 
compared range from personal characteristics and demographics, which are available in records and 
documents (and sometimes from previous surveys), to attitudes and opinions which must be obtained 
from the individuals. 



The results obtained have been inconsistent. Some studies have found significant differences 
(although not all were meaningful differences), others have found none. Even when differences have 
been found in demographics, the variable(s) of interest were not influenced (Finn, Wang & Lamb, 1983). 

The population being surveyed and the survey topic may be mitigating factors which affect the 
need for a high response rate. Some researchers have suggested that a high response rate is less 
important when conducting surveys of homogeneous populations (Becker, Dottavio & Mengak, 1987; 
Becker & Iliff, 1983; Wellman, Hawk, Roggenbuck, & Buhyoff, 1980; Leslie, 1972). McDaniel, Madden and 
Verille (1987) recommended that generalizations about nonresponse must consider the survey topic. 
Brennan and Hoek (1992), however, found patterns in the tendency to respond across topics in two 
surveys of the same population of females over the age of 18. Almost 90% of those who responded to the 
first survey on women and finance also responded to a second survey on attitudes toward and 
participation in survey research. However, more than half of those who did not return forms and just 
over one fourth of those who refused to participate in the first survey did not conform to a pattern and 
chose to participate in the subsequent survey. 

Goudy (1978) and Berdie (1989) have proposed that nonresponse bias be approached from the 
standpoint of representativeness of the respondents and that the appropriate question is whether the 
results would have differed substantially had a higher response rate been obtained. To answer this 
question, then, it is necessary to compare results obtained from early returns with those derived from the 
full set of returns. Differences between response waves are perceived as being relatively unimportant. 
Mail surveys generally obtain their largest response to the initial mailing, thus addition(s) to the data 
from subsequent mailings would have less influence on the cumulative results. Goudy found that "minor 
variations (in variable relationships) appear after approximately 50 percent of the sample have completed 
a mail questionnaire and that no differences exist when about 70 percent of those contacted have returned 
data, especially when bivariate relationships are tested" (p. 264). After re-examining 14 studies to 
compare early returns with total returns, Berdie concluded, "As long as response rates exceed 50 to 60%, 
resources used to promote response rate beyond that level are usually better spent in other ways" (p. 63). 



One homogeneous population that is frequently surveyed consists of college and university 
alumni (Smith & Bers, 1987). Questionnaires are the major approach to obtaining program evaluation 
information from a specific group of alumni, teacher education program graduates (Adams & Craig, 
1983). Response rates in mail follow-up surveys of teacher education program graduates vary from 10% 
to 100%, with a median of 52% (Boser, 1988b). 

In assessing nonrcsponsc bias, college alumni surveys have utilized some of the techniques 
previously described: comparing respondents with a sample of nonrespondents (Carifio & Schwedel, 
1991; Wilkinson; 1976) and with all nonrespondents (Boser 1988a; Hesseldenz, 1976); and comparing early 
versus late respondents or successive response waves (Bowen & Cooper, 1989; Denton, Tsai, & Chevrette, 
1988; Ecklund, 1965; Nielsen, Moos, & Lee, 1978; Pace, 1939). Yet another approach has been to compare 
the results of two successive alumni surveys with considerably different response rates (Hogan, 1985). 

The purpose of the present study is to examine the influence of return rate and sources of 
information other than the survey on the teaching employment rate of individuals in the year following 
completion of the teacher preparation program at a major university. A second objective is to investigate 
the pattern or tendency ^ respond in these individuals. 

Method 

At the University of Tennessee, Knoxville (UTK), the follow-up survey of teacher education 
program graduates is usually conducted in the fall. One of the major purposes of the survey is to 
determine the employment of graduates, which enables the College to provide information to the Career 
Planning and Placement Services on campus about graduates who are still pursuing teaching positions. 

A list of teacher education program graduates was compiled in fall of 1993. Names of individuals 
who participated in the year-long teaching internship in 1992-93 or in a student-teaching experience 
during that school year formed the initial basis for the list. Teacher certification records were cross- 
checked in an effort to eliminate individuals who were not qualifying for initial certification or who were 
applying for licenses in areas other than classroom teaching (e.g., supervisor, counselor, administrator). 
A total of 291 individuals was identified as the target population for the 1993 follow-up survey. This 
group included 221 interns, 26 Lyndhurst fellows, and 44 student teachers. 



At the conclusion of each effort, the number responding, the number and percentage teaching 
that were identified through that effort, the cumulative number and percentage teaching, and the overall 
response rate were calculated. Those categorized as teachers included those employed as full-time 
public or private school teacher, including interim teachers, in a K - 12 setting. 

Chi-square analyses were used for the results at the conclusion of the first, second, and third 
mailings (cumulative), using the cumulative total results as the best estimate for the population to 
determine expected cell frequencies. Chi-square analyses were also conducted comparing the first 
mailing results (early) with the third mailing (late), and with the cumulative results from the three 
mailings (all respondents). Statistical significance was determined using the .05 level. 

Results 

As a result of the initial mailing of the fall survey, 34% of the individuals responded, and 69.8% of 
them were teaching (see Table 1). The second mailing increased the return rate to 51% and the 
cumulative percentage teaching to 71.7%. The third mailing also produced an increase in both response 
rate (57%) and percentage teaching (73.6%). Telephone calls and late questionnaires served to slightly 
depress the teaching employment rate while increasing the response rate by over 20 per cent. When 
information from other sources (public school list, college faculty members, and the spring survey) was 
added, bringing the coverage of graduates to over 93%, the percentage teaching increased to 75.5%. None 
of the chi-square tests yielded statistically significant results. 

Of the 184 who returned questionnaires in the fall survey, 79.3% also returned questionnaires in 
the spring survey (see Table 2). The results were similar for nonrespondents from the fall survey. Of the 
100 who did not return questionnaires in the fall survey, 66.7% did not return questionnaires in the spring 
survey. 

Discussion and Conclusions 
The percentage of graduates employed as teachers changed somewhat with each additional effort 
to locate graduates and obtain information from them. The initial teaching employment rate as a result of 
one mailing (69.8%) was the lowest rate recorded. This represented approximately one third of the 
graduates. A second mailing increased the response rate to over 50% and increased the percentage 



A questionnaire, cover letter, and a postage-paid business reply envelope were mailed to each of 
the individuals on the mailing list in mid- October of 1993 (fall survey). Reminder letters (with 
replacement questionnaires and envelopes) were mailed to nonrespondents three weeks later. A final 
mailing was sent to those who had still not responded after another three weeks (November 29). 
Telephone calls were made in January to those who had still not responded. 

In addition to the survey of graduates, other sources of employment information were used in 
attempts to classify those graduates for whom employment data had not been obtained. A list of new 
teachers in the local school system was obtained, and lists of nonrespondents were sent to UT faculty 
mentoring team leaders. 

A second survey (spring survey) of the same graduates was undertaken in the following spring. 
This questionnaire contained items regarding the nature of their employment, when they were hired, and 
when they thought the follow-up survey should be conducted. One follow-up mailing was sent. 

Seven individuals (four student teachers and three interns) were deleted from the target 
population when it was determined that they had not completed the program and/or licensure 
requirements, thus reducing the target population to 284. No valid address could be found for one 
additional former student. The population was predominantly female (78.2%), and most lived within the 
state (81.3%). Elementary education majors accounted for 51.8% of the population, followed by 
secondary education subject areas (23.6%), technological and adult education (12.3%), special education 
and speech (7.7%), art and music (2.8%), and health and physical education (1.8%). 

A total of 184 individuals responded to the fall survey by mail for a return rate of 65.0%. 
Telephone calls elicited occupational information for 40 individuals. An additional 20 individuals were 
identified as teachers from the local school system list of new teachers, and UT faculty members provided 
employment information for 18 individuals. Three individuals for whom employment information was 
previously unavailable responded to the spring survey. Employment information was initially available 
for 96 individuals who responded to the first mailing (response rate of 33.8%). When all sources of 
information had been utilized, occupations of 265 of the 284 individuals in the target population (93.37o) 
were determined. Ages of the respondents varied from 21 to 56 years with a mean of 26.8 years. 



Table 1 



Fall Survey Results 



hffort 


Respon- 
dents 


By Effort 


Teachers 


Respon- 
dents 


Cumulative 

Teachers 


Response 
Rate 


n 


% 


n % 


Survey of Graduates 














First mailing 


96 


67 


69.8% 




69.8% 


33.8% 


Second mailing 


49 


37 


75.5% 


145 


104 71.7% 


51.1% 


Third mailing 


18* 


16 


88.9% 


163 


120 73.6% 


57.4% 


Phone 


40 


29 


72.5% 


203 


149 73.4% 


71.5% 


Late questionnaires 


21** 


13 


61.9% 


224 


162 72.3% 


78.9% 


Other Sources 














Public school list 


20 


20 










College faculty 


18 


15 










Spring survey 


3 


3 










Total 








265 


200 75.5% 


93.3% 



^Includes one response by letter 

^Includes 8 questionnaires received after telephone contact (not included in telephone count) 



teaching by less than two percent. The highest recorded percentage of teachers derived from the returned 
questionnaires (73.6%) occurred after the third mailing and represented 57.4% of the graduates. When 
the response rate was increased by means of telephone contacts and late questionnaire. , the percentage 
employed as teachers declined slightly. The difference in cumulative percentage teaching between the 
early returns (first mailing) and any stage up to and including the total survey returns (78.9%) never 
exceeded 3.8%. Only by consulting sources other than the graduates (public school list and college 



Table 2 
Tendency to Respond 





Fall Survey 




Spring Survey 




Cumulative 




Response 


Results 




Respondents 


Fall 


Spring 




Group 


n 


n 


% 


n 


n 


% 


Mail Respondents to 
Fall Survey 














First wave 


96 


78 


81.3% 








Second wave 


49 


40 


81.6% 


145 


118 


81.4% 


Third wave 


18* 


12 


66.7% 


163 


130 


79.8% 


Late 


21 


16 


76.2% 


184 


146 


793% 


Nonrespondents to 
Fall Survey 














Phone follow-up 


40 


13 


32.5% 








Did not respond 
(Faculty, public school 
list, no information) 


60 


20 


33.3% 


100 


3: 


33.3% 



'Includes one individual who responded by letter to fall survey. 



faculty members) was the employment rate substantially impacted. The use of the public school list of 
new employees introduces a bias, however, because it provides identification for only those graduates 
who arc known to be teaching and employed in one specific school system. There is no similar source of 
information about those not teaching. 

The actual teaching employment rate falls somewhere between 70.4% and 77.1%, Those figures 
are calculated as extremes. If none of the 19 for whom employment was known were teaching, the final 
number of teachers would have been 200 (out of 284, 70.4%); if all 19 were teaching, the final count would 
have been 219 (77.1%). All cumulative percentages of teachers fall within these bounds except the results 
from the first mailing (69.8%), at which point the response rate was 33.8%. 
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Comparison of early returns (first mailing) with later returns (third mailing) would have shown a 
much different picture, with a discrepancy of almost 20 per cent between the first mailing (69.8%) and the 
third mailing (88.9%). However, the largest number of individuals responded to the first mailing, and 
only approximately 20% of that number responded to the third mailing, hence the impact of adding those 
returns was lessened considerably. It is important to note that the difference between the first and third 
mailings was not statistically significant, calling attention to the need for information by consumers of 
research. If only the percentages of teachers were known, readers might easily conclude that those who 
responded to the first mailing were indeed different in a practical sense from later respondents (who 
might be considered as representing the nonrcspondents) even though statistical tests were not 
supportive. Recognition of the considerable difference in group sizes is important in this instance. 

Following the reasoning of Goudy (1978) and Bcrdic (1989), comparing the results of the first 
mailing with the cumulative total results and with the cumulative results after the three mailings showed 
that from a statistical standpoint, the early returns did not differ significantly from those achieved by 
subsequent efforts. This approach presents a much less striking picture of potential bias than the 
previously described comparison of the employment rates between respondents to the first and third 
mailings. Overall, increases in the response rate from 34% to 57% and obtaining information from other 
sources so that employment of 93% of the individuals was known had a fairly small but observable effect 
on the teaching employment rate. Whether or not the actual difference (less than 6%) is important 
enough to justify the expenditure of time and resources to obtain the additional information is a matter 
for the researcher to decide. It seems apparent that the preferred method of examining nonresponse bias 
is to compare early respondents with all respondents, considering one third of the individuals as a 
minimum for the early respondent group, in this or similar surveys of homogeneous populations. This 
does not in any way suggest that mail survey researchers should discontinue their efforts to achieve high 
response rates. This is one specific endeavor to determine a particular population characteristic (teaching 
employment) for a homogeneous target population (teacher education graduates). 

In looking for patterns showing a tendency to participate or not participate in the mail surveys 
for individuals, comparison of participation in the fall and spring surveys yielded results very supportive 
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of Brennan and Hoek (1992). Almost 80% of those who responded to the fall survey returned 
questionnaires in the spring survey, not quite as high as the nearly 90% found by Brennan and Hoek . Of 
those who did not participate in the first survey, the 60% who also elected not to participate in the spring 
survey in the current study is slightly higher than the approximately 54% found by Brennan and Hoek. It 
appears that deciding to participate in a survey, at least for college surveys of teacher education alumni, is 
highly dependent on the inclination of the individual. 
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